Comparing Usability Problem Identification and Description by Practitioners and Students

Many studies of usability studies count the number of usability problems identified to measure the effectiveness of an evaluation. However, communicating problems is also important to evaluation effectiveness, because a problem found but poorly explained may not be fixed. This study compared lists of usability problems from 21 practitioners and 23 students watching a pre-recorded usability session. Lists were evaluated for the number of problems reported, and for following six guidelines for describing usability problems: be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, and provide a solution. There was no difference in the number of problems reported by students and practitioners, but there was a difference in their ratings for following several of the guidelines. Using both measures provides a more complete assessment of usability reports.